An Assessment Framework for DialPort

Collecting a large amount of real human-computer interaction data in various domains is a cornerstone in the development of better data-driven spoken dialog systems. The DialPort project is creating a portal to collect a constant stream of real user conversational data on a variety of topics. In order to keep real users attracted to DialPort, it is crucial to develop a robust evaluation framework to monitor and maintain high performance. Different from earlier spoken dialog systems, DialPort has a heterogeneous set of spoken dialog systems gathered under one outwardlooking agent. In order to access this new structure, we have identified some unique challenges that DialPort will encounter so that it can appeal to real users and have created a novel evaluation scheme that quantitatively assesses their performance in these situations. We look at assessment from the point of view of the system developer as well as that of the end user. *Kyusong Lee Carnegie Mellon University, PA, USA e-mail: [email protected] *Tiancheng Zhao Carnegie Mellon University, PA, USA e-mail: [email protected] Stefan Ultes Cambridge University, Cambridge, UK e-mail: [email protected] Lina Rojas-Barahona Cambridge University, Cambridge, UK e-mail: [email protected] Eli Pincus University of Southern California, CA, USA e-mail: [email protected] David Traum University of Southern California, CA, USA e-mail: [email protected] Maxine Eskenazi Carnegie Mellon Unversity, PA, USA e-mail: [email protected] *Both authors contributed equally to this paper